Solved: PySpark: Writing Parquet Files to the Azure Blob S... |
您所在的位置:网站首页 › databricks read parquet file from s3png › Solved: PySpark: Writing Parquet Files to the Azure Blob S... |
Currently I am having some issues with the writing of the parquet file in the Storage Container. I do have the codes running but whenever the dataframe writer puts the parquet to the blob storage instead of the parquet file type, it is created as a folder type with many files content to it. One note to it, I tried searching various ways in the internets that it seems this is default creation when using pyspark and I can see in the folder created there was a file parquet with a snappy add into it (refer to the screenshots below) If this is a default creation to pyspark code, how can I write a single parquet format that will do some splitting or creation of folder? any recommendations? on how to do it? |
CopyRight 2018-2019 办公设备维修网 版权所有 豫ICP备15022753号-3 |